feat: TEAV JDBC insert/update [DHIS2-21378]#24098
Merged
Merged
Conversation
|
ameenhere
approved these changes
Jun 11, 2026
muilpp
approved these changes
Jun 11, 2026
enricocolasante
added a commit
that referenced
this pull request
Jul 7, 2026
… (2.43) (#24286) * feat: remove EntityManager from TrackerPersister [DHIS2-21378] (#23797) * feat: remove EntityManager from TrackerPersister interface and method parameters [DHIS2-21378] * Use default timezone (cherry picked from commit 052d3ac) * feat: decouple ChangeLogAccumulator from Hibernate Session [DHIS2-21378] (#23836) (cherry picked from commit 0c1779f) * chore: Extract FileResource assignment update [DHIS2-21378] (#23879) * chore: Extract FileResource assignment update [DHIS2-21378] * Code review fixes * Code review fixes (cherry picked from commit 83542f4) * feat: stage TEAV writes through EntityWriteBatch [DHIS2-21378] (#23970) (cherry picked from commit b30c1ce) * feat: Batch entity when persisting [DHIS2-21378] (#24005) (cherry picked from commit 100dff1) * feat: remove EntityManager from DefaultTrackerBundleService [DHIS2-21378] (#24024) (cherry picked from commit 3759453) * feat: TrackedEntity JDBC insert [DHIS2-21378] (#24027) (cherry picked from commit 4eaf6ba) * feat: TrackedEntity JDBC update [DHIS2-21378] (#24079) (cherry picked from commit d4af287) * feat: Enrollment JDBC insert/update [DHIS2-21378] (#24081) * feat: TrackedEntity JDBC update [DHIS2-21378] * feat: Enrollment JDBC insert/update [DHIS2-21378] * Remove L2 caching from enrollment * Code review fixes * Code review fixes (cherry picked from commit 34d629e) * feat: Tracker event JDBC insert/update [DHIS2-21378] (#24082) * feat: TrackedEntity JDBC insert [DHIS2-21378] * feat: TrackedEntity JDBC update [DHIS2-21378] * feat: Enrollment JDBC insert/update [DHIS2-21378] * feat: Tracker event JDBC insert/update [DHIS2-21378] (cherry picked from commit 435872b) * feat: Single event JDBC insert/update [DHIS2-21378] (#24083) * feat: TrackedEntity JDBC insert [DHIS2-21378] * feat: TrackedEntity JDBC update [DHIS2-21378] * feat: Enrollment JDBC insert/update [DHIS2-21378] * feat: Tracker event JDBC insert/update [DHIS2-21378] * feat: Single event JDBC insert/update [DHIS2-21378] * Format code (cherry picked from commit 9703e4f) * feat: Relationship JDBC insert/update [DHIS2-21378] (#24093) * feat: TrackedEntity JDBC insert [DHIS2-21378] * feat: TrackedEntity JDBC update [DHIS2-21378] * feat: Enrollment JDBC insert/update [DHIS2-21378] * feat: Tracker event JDBC insert/update [DHIS2-21378] * feat: Single event JDBC insert/update [DHIS2-21378] * feat: Relationship JDBC insert/update [DHIS2-21378] (cherry picked from commit 658f0a8) * feat: TEAV JDBC insert/update [DHIS2-21378] (#24098) * feat: TrackedEntity JDBC update [DHIS2-21378] * feat: Enrollment JDBC insert/update [DHIS2-21378] * feat: Tracker event JDBC insert/update [DHIS2-21378] * feat: Single event JDBC insert/update [DHIS2-21378] * feat: Relationship JDBC insert/update [DHIS2-21378] * feat: TEAV JDBC insert/update [DHIS2-21378] * Code review fixes * Fix tests * Fix idScheme issue for TEAV (cherry picked from commit 4e3367c) * fix: Refactor persistence layer in tracker importer [DHIS2-21378] (#24197) * feat: TrackedEntity JDBC update [DHIS2-21378] * feat: Enrollment JDBC insert/update [DHIS2-21378] * feat: Tracker event JDBC insert/update [DHIS2-21378] * feat: Single event JDBC insert/update [DHIS2-21378] * feat: Relationship JDBC insert/update [DHIS2-21378] * feat: TEAV JDBC insert/update [DHIS2-21378] * Code review fixes * Fix tests * Fix idScheme issue for TEAV * fix: Refactor persistence layer in tracker importer [DHIS2-21378] * Fix merge (cherry picked from commit 194dcd1) * fix: Harmonize db sequences for tracker [DHIS2-21378] (#24208) * fix: Harmonize db sequences for tracker [DHIS2-21378] * Fix relationshipitemid (cherry picked from commit f52eefa) * fix: Rename conflicting migration [DHIS2-21378] (#24218) (cherry picked from commit 7aefc1c) No-op on 2.43: this PR renamed V2_44_12 -> V2_44_13 to avoid a master-only Flyway version clash. On 2.43 the migration is numbered V2_43_63 (its own series), so no rename is needed. Empty commit kept to preserve the full DHIS2-21378 backport history. * refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] (#24227) * refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] * chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] * Code review fixes (cherry picked from commit 331503a) * chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] (#24239) * refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] * chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] (cherry picked from commit 7ba2431) * perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378] (#24244) * refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] * chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] * Code review fixes * perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378 * Code review fixes * Code review fixes (cherry picked from commit 2972a82) * perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378] (#24250) * refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] * chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] * Code review fixes * perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378 * perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378] * perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378] * Revert "perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]" This reverts commit 7084379. * Add tests to cover fields import (cherry picked from commit 7305f9f) * perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378] (#24252) * refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] * chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] * Code review fixes * perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378 * perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378] * perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378] * Fix duration based Tracker perf tests * Fix formatting (cherry picked from commit a03204c) * fix: Preserve storedby on tracker JDBC writers for 2.43 [DHIS2-21378] Backport adaptation. On master, DHIS2-21537 removed storedBy from the TrackedEntity/Enrollment/Event write path (and renamed TEAV storedBy to updatedBy). 2.43 deliberately keeps storedBy, so the JDBC writers backported from master must persist the storedby column that 2.43's tables still have. TrackerObjectsMapper still sets storedBy on the new-entity branch for TE, enrollment and events, so storedby is insert-only here (mirrors createdby): added the storedby column + bind to the INSERT of TrackedEntityWriter, EnrollmentWriter, SingleEventWriter and TrackerEventWriter. TeavWriter was already adapted (storedby column + getStoredBy) when the earlier commits were cherry-picked. Keeps the 2.43-only shouldSetStoredByToAuthenticatedUser* tests green. * Fix note persister * Fix formatting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final slice of the tracker importer Hibernate→JDBC write-path migration DHIS2-21378: moves
TrackedEntityAttributeValue(TEAV) INSERT/UPDATE/DELETE from theEntityManagerto JDBCbatching in
EntityWriteBatch, and removes theEntityManagerfrom the persister layerentirely. With this PR, every write in the tracker import commit phase goes through JDBC on the
transaction-bound connection — no Hibernate-managed entities are written during commit, so the
L1 refresh/detach reconciliation that earlier phases needed is deleted too.
Changes
Write path (
dhis-tracker)EntityWriteBatch: TEAVs are now flushed via JDBC againsttrackedentityattributevalue— amulti-row INSERT, a single
unnestUPDATE keyed on the composite (trackedentityid,trackedentityattributeid) PK (createdexcluded, insert-only), and a singleunnestDELETEon the same key. No id pre-allocation: the table has a composite PK and no sequence. Values are
written as plain text (confidential attributes and
encryptedvaluewere removed in DHIS2-21518).EntityWriteBatch.flush(Connection)no longer takes anEntityManager; therefreshUpdatedTrackerEvents/refreshUpdatedSingleEventsL1 reconciliation is removed — thereare no managed instances left to reconcile.
AbstractTrackerPersister: the@PersistenceContext EntityManagerfield, the JPQL "existingTEAVs" lookup, and the per-flush
entityManager.flush()calls are gone. Existing attributevalues are read with a direct JDBC query (
loadExistingAttributeValues) on the sametransaction-bound connection the writes use, producing plain never-managed
TrackedEntityAttributeValueinstances — so a TEAV written by a prior persister in the sameimport (e.g. the same attribute on a TrackedEntity and on an Enrollment payload) is found and
routed to an UPDATE instead of a duplicate INSERT.
UID — the preheat maps are keyed by the import's configured idScheme, so a UID-based resolution
would silently miss every existing value under
idScheme=CODE/NAME/ATTRIBUTEand turn updatesinto composite-PK violations.
Connectionis threaded throughupdateAttributes(all five persisters)instead of re-acquiring a connection per tracked entity.
persistOwnershipjavadoc documents the one remaining Hibernate write (TrackedEntityProgramOwneron enrollment) and its failure-attribution trade-off: a constraintviolation (concurrent-import race) surfaces at commit, outside the per-entity try/catch.
@Auditable(scope=TRACKER), and JDBC writes bypass Hibernate's auditlisteners — consistent with the other migrated entity types. TRACKER-scope auditing is disabled
by default, and attribute-value history is covered by the
trackedentitychangelogtable.Tests
startNewRequestSession()helper (flush + clearbetween imports): production gives each
/trackerrequest a fresh transaction-scoped session,while MockMvc tests share one thread-bound session — since the importer writes via JDBC, a
stale managed entity would otherwise leak into the next import's preheat.
(
manager.clear()/dbmsManager.clearSession()).TrackerEventSMSTest: uses a second-precision occurred date — SMS encoding drops milliseconds,which the previous Hibernate write path masked.
Performance
Compared against
master-without-DHIS2-21378(GH Actions "Performance tests compare" runs27197625605 alloc,
27202162811 cpu,
27202229361 wall, 2026-06-09).
Note the baseline excludes the whole migration, so these numbers measure the cumulative
DHIS2-21378 win with this branch as the final state — not this PR's delta alone.
-e alloc)-e cpu)-e wall)Drivers: Hibernate flush/dirty-check/snapshot machinery eliminated (alloc −2.9 GiB, CPU −2127
samples), per-entity statements collapsed into multi-row INSERTs /
unnestUPDATEs (DB socketwait 1554 → 559 samples; jsonb now serialized once instead of deep-copied + compared per flush).
Phase-level breakdown shows the commit phase is now ~90% irreducible DB write time — remaining
opportunities are all read/validation-path and belong in separate tickets.
Verdict: large improvement; the write path is at its structural floor.
Next Steps
Done
ChangeLogAccumulator,FileResource store extraction, TEAV staging, entity staging,
lastupdatedbulk update(feat: remove EntityManager from TrackerPersister [DHIS2-21378] #23797, feat: decouple ChangeLogAccumulator from Hibernate Session [DHIS2-21378] #23836, chore: Extract FileResource assignment update [DHIS2-21378] #23879, feat: stage TEAV writes through EntityWriteBatch [DHIS2-21378] #23970, feat: Batch entity when persisting [DHIS2-21378] #24005, feat: remove EntityManager from DefaultTrackerBundleService [DHIS2-21378] #24024)
TrackerEvent (feat: Tracker event JDBC insert/update [DHIS2-21378] #24082), SingleEvent (feat: Single event JDBC insert/update [DHIS2-21378] #24083), Relationship (feat: Relationship JDBC insert/update [DHIS2-21378] #24093)
EntityManagerremovalTo do
relationship/relationshipitem(both currently draw from theshared
hibernate_sequence;RelationshipItem.idis anint, so the global counter riskssilent narrowing)
EntityWriteBatchacross persisters, making the batch the source of truth for staged TEAVs and dropping the per-TE existing-values round-trip (TODO inhandleTrackedEntityAttributeValues)EntityWriteBatch